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Education and Learning (McREL) . During 1999, the fourth year of the 
Laboratory's contract period with the Office of Educational Research and 
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impact evaluation. Needs assessment, formative, and summative evaluations, 
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Nationwide Education Information & Assistance System; (5) evaluation of the 
Laboratory network program; (6) assistance to the Office of Educational 
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evaluation study. Each of these studies is described, with a description of 
the specific focuses of component studies that make up the nine major 
studies. Eleven appendixes contain outlines of some aspects of these 
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FOREWORD 



This evaluation plan is revised and updated annually. It was initially prepared in September, 
1995 in support of McREL's proposal for the Central Regional Educational Laboratory (RFP-95- 
040), refined in view of the final contract statement of work and internal project planning in May, 
1996, and subsequently revised in March, 1997. The plan, which draws substantially from the 
Laboratory's plan for the previous five-year contract period, provides a synthesis of the various 
evaluation activities. Given the evolving nature of this document, several activities, specifically 
those associated with special evaluation studies are determined annually. During 1999, the fourth 
year of the contract period, special studies and integrated studies were designed to bridge the fourth 
and fifth years in order to increase the emphasis on summative or impact evaluation. 



INTRODUCTION 



In recent years, the need for enhanced accountability and demonstrated impact of our nation's 
social interventions has been fueled by Congress' increased attention to budgets, deficits, and 
national trends that reflect the social, economic and educational needs of all citizens. Concerns over 
our standing in world markets have been traced to the very core of our educational system. Now 
more than ever before there is a need to re-assess the fundamental assumptions and principles that 
have guided the initial development and evolution of the nation's education system. This need for 
re-thinking effective school design does not imply that all current approaches to education must be 
replaced simply because they now exist. Rather, those aspects of the system that are working well 
and serve to prepare students for the future must be identified and replicated and those components 
that are ineffective and are likely to impede progress toward school and system redesign must be 
modified or replaced. A combination of research and development and evaluation can support and 
inform the difficult decisions that must be made in order to improve the overall design of the 
educational process. Research and development can supply the theoretical base and alternative 
intervention strategies that can then be tried and tested in schools using carefully crafted evaluations. 

In response to these needs the national Regional Educational Laboratory network was 
established by the Department of Education and represents a substantial investment of resources. 
Therefore, it is essential that the impacts and accountability of the network be monitored and 
documented. It is important to know how effective the network is in stimulating positive changes, 
to understand how the changes have been achieved, and the extent to which the government's 
resources are being effectively leveraged. These data must be derived from the individual 
laboratories, which in turn must build their overall evaluations on the assessment of individual 
program components and services delivered. This document describes how McREL intends to 
provide these data. 

Evaluation is an integral component of McREL's Central Regional Educational Laboratory, 
one that pervades every major task and serves to assess the progress and impact of the Laboratory, 
as well as guide its direction over the five-year contract period. McREL continues its commitment 
to conducting evaluations and using results through this formal plan for embedding ongoing 
evaluation and feedback into all laboratory operations and services. 



GOALS OF EVALUATION 

Collectively, the general evaluation goals outlined in the RFP address the need for 
accountability, assurance of quality for all products and services delivered, on-going regional needs 
assessment, documentation of processes and activities used to attain outcomes, and assessment of 
impacts on the region's educational systems. Thus, the primary goal of the evaluation is 



to determine what does and what does not work well in improving the 
educational process and student learning throughout the region, to evaluate the 
effectiveness of the Laboratory in developing sustained local capacity for self- 
assessment and systemic change, and to assess the extent to which efforts to scale- 
up systemic reform have been successful. 

In view of the evaluation requirements for Regional Educational Laboratories established by 
OERI, and our own plan for using evaluation data to guide laboratory operations, the following 
objectives are identified for this evaluation: 



1 




7 



• to demonstrate that the investment in the regional laboratories network is a 
strategy that effectively leverages the government's funds in producing impacts 
on the educational system and its students; 

• to demonstrate that the laboratories, collectively and in collaboration with other 
agencies, are effectively contributing to the process of improving the nation's 
educational system through systemic reform; 

• to determine if activities and services of the laboratory are addressing the needs 
of the region's educational system, public, and students; 

• to assess the extent to which the laboratory is operating as intended/planned and 
is sensitive and responsive to changing regional needs; 

• to assure that the products and services of McREL are of the highest quality; 

• to determine the extent to which the R&D projects conducted meet the highest 
standards of design and methodological rigor, and help fill current gaps in 
knowledge and practice; 

• to determine the extent to which R&D findings are effectively integrated into 
service and collaborative components of laboratory functioning; and 

• determine the extent to which McREL contributes to the scaling9-up priority of 
OERI within the region and across the nation through participation in a 
nationwide education information and assistance system. 



EVALUATION APPROACH 

The evaluation staff works closely with McREL's Executive Director, project directors, and 
members of appropriate advisory groups in order to develop evaluation efforts that are well planned, 
carefully integrated, and implemented in a manner that augments the primary objective of supporting 
and serving the region's educational needs. Recognizing the exploratory nature of some laboratory 
activities, the role of evaluation in assisting and informing the service delivery process is 
emphasized. From a program perspective, for example, a particular strategy, service, hypothesis or 
research effort is considered successful if it exposes a common misunderstanding, even though the 
strategy employed may have been unsuccessful in achieving expected outcomes. 

A Collaborative Approach 

McREL employs a collaborative approach to evaluation that requires the active cooperation 
of all stakeholders: laboratory management, service providers, clients, and the evaluation team. 
While the traditional notions of objectivity and independence are certainly important features of 
quality evaluations, it is also important to recognize that a lack of communication among clients, 
service providers, and evaluation staff can also have negative consequences. Aside from the 
adversarial working relationship created, evaluators would not have access to field workers' insights 
that help explain how observed impacts were attained. Without such input it is more likely that 
objectives, processes, and impacts would be misinterpreted. Further, with only limited or periodic 
contact, project directors would not receive the benefit of evaluation findings in a timely and useful 
manner which could result in wasted resources and decreased project success. 
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McREL's experience with the collaborative evaluation concept suggests that the cooperative 
spirit does much to enhance the utility of evaluations and ultimately the impact of program 
interventions and services. Cooperation between service delivery and evaluation staff does not 
necessarily preclude objectivity. On the contrary, more reliable and objective information is often 
obtained since all involved persons have a stake in producing accurate, timely, and useful 
information. In accordance with the concept of collaborative evaluation, the McREL evaluation and 
service delivery staffs works together to refine objectives, select measures of performance, collect 
data, interpret findings, and communicate conclusions. 

Planning Process 

The development of specific evaluation studies and the periodic review and revision of the 
total plan during the five-year project period is guided by a number of considerations: flexibility, 
practicality, cost-effectiveness, and utility. Without question, the conduct of evaluation studies 
consumes considerable project resources. By viewing the laboratory project as an integrated series 
of related tasks and activities, it is possible for McREL to assess the evaluability of various 
components of the project before significant resources are expended. Of particular concern is the 
feasibility of conducting an evaluation given the level of commitment of the partner(s) involved, the 
data likely to be available, and the accessibility of required data. Closely related to these practical 
considerations are the issues of cost and utility. McREL's evaluation planning process includes a 
preliminary assessment of the cost of the evaluation effort in relation of the utility of the findings. 

Evaluation Framework 



In order to effectively address the diverse range of evaluation goals, this evaluation plan has 
been organized around four levels of laboratory and cross-organizational functioning rather than 
according to the type of evaluation being conducted. This organization helps assure that each 
evaluation effort is designed and implemented in view of its purpose and utility, and not driven 
solely by methodological concerns. Figure 1 depicts McREL's evaluation framework. As shown 
in the figure, levels are aligned in a "bottom-up" fashion with each successive level encompassing 
increasingly general aspects of laboratory functioning while addressing needs assessment, formative, 
summative, and quality assurance issues. Level I assessments target individual activities within tasks 
or projects, such as a particular service intervention strategy. Level II evaluations examine the 
operation and outcome of each major task (management, development and applied research, field 
services, etc.) and its associated projects. Level III evaluations focus on the processes and outcomes 
of the Laboratory as a whole and on cross-task themes. Finally, Level IV examines collaborative 
efforts among the laboratories and other OERI-sponsored organizations. 



Level IV 



Level III 



Level II 
Level I 




Figure 1. McREL’s overall evaluation framework 
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This framework is designed to facilitate the management of the evaluation work and increase 
the efficiency of data collection. Note that the relationship among the levels is overlapping but not 
entirely hierarchical. That is, while evaluations at each level are expected to benefit from the 
information available through lower levels of the framework, it is also expected that some new, 
level-specific information will be needed. Primary data will enter the evaluation structure through 
a series of integrated, yet independently implemented evaluation studies. Over the course of the 
contract period, this cumulative approach results in an integrated assessment of overall laboratory 
functioning and impact. 

McREL believes that organizing the evaluation effort into specific levels also makes it easier 
to decide which aspects of the laboratory program are most suitable for in-depth evaluation. Using 
this approach allows McREL to incorporate new evaluation components as new services or R&D 
projects are initiated, and to delete old evaluation efforts as they are completed or as nonproductive 
projects are discontinued. This flexibility in the evaluation plan is essential given the dynamic nature 
of the Laboratory and the importance of addressing changing regional needs over a five-year period. 
It must be possible to accommodate changes in direction and focus without negatively influencing 
other aspects of the ongoing evaluation effort. 

Level I . At this level, individual activities conducted under each task will be assessed. That 
is, the adequacy of a particular technical assistance or staff development event, or a specific R&D 
planning effort might be assessed. These evaluations will emphasize formative issues. The results 
of these formative evaluations will be used to continually improve how services are planned and 
delivered and how R&D projects are designed and implemented. As such, the primary users of these 
findings will be Laboratory staff. Impacts of these assessments on Laboratory functioning are 
expected to include increased efficiency, timeliness, and user satisfaction with respect to the 
Laboratory services and products developed and delivered. Evaluation at this level relies primarily 
on the established infrastructure of the Laboratory that is designed to routinely collect data regarding 
staff activities. 

Level II . This intermediate level of evaluation is designed to target the individual 
components of the Laboratory operation. Thus, it will be possible to determine which aspects of the 
program are responsible for the observed impacts of Laboratory operations. This level of evaluation 
will examine separately each of the primary tasks defined for the Laboratory contract on an annual 
basis. Specifically, in Task 1 , Laboratory Management, evaluation plays a key role in assessing the 
role of the Board of Directors in directing the Laboratory, staff perceptions of Laboratory operations, 
and the needs of educators in the region. Further evaluation findings and conclusions serve as input 
for guiding management, planning, and redirection process as defined by the governing board. In 
Task 2, Development and Applied Research, evaluation is used to support ongoing R&D projects 
and to build a capacity for evaluation among local partners. In Task 3, Field Services, evaluation 
is used to assure quality products and services and to assess the impacts of services on McREL's 
clients. Needs assessment in support of service planning is also conducted. In addition to the overall 
evaluations conducted in relation to both Tasks 2 and 3, special studies, organized around the 
individual service intervention sites and R&D projects, are conducted on an ad hoc basis. These 
special studies will focus on selected topics related to major R&D or service initiatives, specific 
types of interventions, overall impacts in targeted states, or on the assessment of particularly 
innovative interventions that might be suitable for replication in other locales. 

In support of Task 4, Participation in the Nationwide Education Information & Assistance 
System, McREL's evaluation team collects and analyzes data in support of system development and 
operation. In Task 5, the Laboratory Network Program, the evaluation team collaborates with 
evaluators from the other laboratories to better understand cross-laboratory projects and, in 
particular, support the continued development of a performance indicator system that can be applied 
network-wide. In Task 6, Assistance to OERI, staff conducts short-response time evaluation 
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activities in response to OERI requests. Finally, in Task 7, Specialty Area Development, evaluation 
plays a supporting role in determining the ability of the Laboratory to integrate and disseminate high- 
quality, useful information in an efficient and effective manner. 

Level III . At the overall Laboratory program level, issues of impact within the region and 
the processes used to achieve them are the targets of evaluation. To a great extent, evaluation at this 
level relies on the data collected in the task and activity evaluations discussed above. The primary 
difference between levels is in how the data are analyzed and reported. In the earlier years of the 
contract, the overall Laboratory evaluation focused, of necessity, on formative evaluation issues. 
However, as sufficient time passes to allow anticipated outcomes to manifest themselves, it is 
possible to examine the Laboratory performance in terms of specific impacts across the region. 

Because needs assessment has and will continue to guide the selection of laboratory service 
topics and R&D studies conducted over the contract period, regional needs assessment are conducted 
informally on a continuous basis and formally on a periodic basis. Because needs assessment 
influences the way in which the Laboratory relates to the region and its perceived needs, it is 
emphasized at Level III of the evaluation plan. 

Level IV . It is most important that the Regional Educational Laboratories and other OERI- 
funded entities share information about the successful and unsuccessful processes used to deliver 
services and translate R&D findings into practice and policy. McREL recognizes that participation 
in the OERI coordinated cross-laboratory evaluation must be included as a component of its 
evaluation plan. The objectives of this coordinated evaluation effort are as follows: 

• to share information about which laboratory services and products are most 
effective in attaining stated educational outcomes, 

• to identify which strategies can be made more effective based on the collective 
experiences of the Regional Educational Laboratory network, and 

• to assess the overall appropriateness and effectiveness of the regional laboratory 
concept in stimulating improvements in the educational process. 

At the cross-organizational level, McREL participates in evaluations that are jointly designed 
and implemented with other laboratories, centers, and partner organizations. These evaluations go 
beyond the immediate information needs of the Laboratory and more directly address the 
programmatic needs of OERI. Task 4, Participation intheNEIA System, and Task 5, the Laboratory 
Network Program and the performance indicator system, invoke this level of evaluation, although 
work in other areas may also prove to be relevant given McREL's emphasis on collaboration in its 
five-year statement of work. 

Evaluation Methods and Strategies 

Needs assessment, formative and summative evaluations and quality assurance activities are 
embedded throughout McREL's five-year plan of work, although not necessarily as independent 
evaluation studies. The relative emphasis on each type of evaluation varies substantially depending 
on the information required and the activity in question. In general, needs are assessed both 
informally on an ongoing basis through our contacts with educators and service providers and 
formally through periodic surveys and discussion groups. This work is designed to ensure that the 
work of the Laboratory is consistent with the identified needs of the region. Formative evaluations 
address issues of context, the quality and quantity of inputs, and processes employed. Formative 
evaluations serve two informational needs: (a) they provide immediate feedback to project personnel 
and the Board of Directors about how the Laboratory or its components are operating, and (b) they 
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provide an understanding of how observed project impacts/outcomes were attained. Summative 
evaluations are conducted at both the laboratory and individual task levels. These studies focus on 
identifying the outcomes and impacts of activities, and occasionally address the adequacy of impacts 
attained in relation to local, state, or regional needs. 

Although the data collection methods used for each particular evaluation study are determined 
by the particular information requirements and available resources, most efforts rely on multiple 
sources of information. Figure 2 provides an overview of these sources, as they relate to three broad 
categories. First, an evaluation infrastructure is maintained by staff activity reports and supported 
by a variety of supplemental data such as information request logs, World Wide Web statistics, and 
participant feedback. Second, existing information from program and project files and other 
resource collections is utilized. This information comes from project reports, the professional 
literature, or national databases such as those maintained by the National Center for Educational 
Statistics. Third, supplemental data collection efforts are conducted as necessary. Typically, these 
involve surveys, interviews, and on-site data collection techniques. 



OVERVIEW OF EVALUATION STUDIES 

Table 1 provides an overview of the evaluation studies' planned for the entire five-year 
contract period, including those completed to date. Each of the special studies outlined is fully 
documented with a separate planning report, and final report of findings or a section in the Annual 
Evaluation Report. 

Evaluation Infrastructure 



An internal McREL Evaluation Guide has been prepared by the evaluation team which 
describes the planning process, data collection forms and procedures, reporting formats, and 
archiving requirements associated with all evaluation work. As changes are made in the evaluation 
infrastructure they are documented in the Guide. Professional development also plays a key role in 
strengthening McREL's evaluation infrastructure by including field staff directly in the assessment 
activities. A detailed review and revision of the Guide was conducted in Year 3. Revisions are 
planned in Year 5 to enhance the internal review aspects of the quality assurance process for 
evaluation documents. 

Evaluation of Laboratory Management (Study 1) 

This evaluation of Laboratory management and operations is based on the integration of the 
assessments of three key elements: board operations, project management, and needs assessment 
processes. This study not only presents the activities and findings on an annual basis, but also 
includes a comparative analysis of changes made as a result of the previous year’s findings. It is 
designed to include an examination of the activities of the Board of Directors; its structure, role, 
operating procedures, and impact on Laboratory operations and mission. All Board meeting 
documents are reviewed to determine action items required and general areas of focus for service 
and R&D efforts to be pursued. Ongoing activities of the Laboratory are then examined in light of 
this information. Of particular interest is the consistency between Laboratory activities and the 
direction provided by the Board of Directors and the Executive Director. The analysis of Laboratory 
management structures and practice is based on a content analysis of management and staff meeting 



*The term “study” is used to denote associated evaluation tasks so as not to confuse them with the seven laboratory t<u>ks. In some 
instances, these evaluation activities actually involve support or technical assistance rather than investigation as the term study implies. 
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ACTIVITY REPORTING 



DOCUMENTATION 



► Agenda 

► Products 

► Supporting Materials 

COMMUNICATION 

► www statistics 

► E-mail 

► Media 



PUBLICATION 



► Distribution 

► Orders 



PARTICIPATION 



► Registration 

► Client Profiles 



ASSISTANCE 



► Request log 

► Resources circulation 



FEEDBACK 



► Evaluations 

► Staff debriefing 

► Support letters 

► Suggestion box 

► Awards/recognition 

PROGRAM & PRODUCT 
INFORMATION 



► Staff interviews Professional literature 

► Work plans Data bases 

► Reports Success stories 

► Products 

STUDY-SPECIFIC METHODS 

► Interviews Assessment 

► Questionnaires Focus groups 

► On-site observations Educational records 

► Peer/expert panels Tracer studies 

Figure 2. Sources of evaluation data 
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Table 1 



Summary of Planned and Completed Evaluation Studies 



LEVEL 


STUDY 


1 


CONTRACT YEAR 
2 3 4 


5 


I 


Event and Activity-Specific Evaluations 


yf 


y/ 


V 


yf 


y 


II 


1 . Laboratory Management 


yf 


y/ 


y/ 


yf 


y 




1.1 Board of Directors survey 




y/ 


4 


yf 


y 




1 .2 Regional Needs Assessment 






4 




4 




1 .3 Staff Perceptions of Organizational 


yf 


y/ 


y/ 


yf 


y 




2. Development and Applied Research 


yf 


y/ 


V 


yf 


y 




2. 1 Capacity Building through Collaborative 










y 




3. Field Services 


yf 


y/ 


yf 


yf 


y 




3.1 Publications Evaluation 


v/ 


4 




yf 






3.2 Assessment of Training Impacts 


v/ 












3.2.1 A+ Training in Nebraska 


yf 












3.2.2 SDDB Follow -Up 


y/ 












3.2.3 Colorado BOCES 




y/ 










3.2.4 Compendium Follow-Up 




yf 


4 








3.2.5 Client Satisfaction 






y/ 


yf 






3.2.7 RIP Series Study 








4 


y 




3.2.8 CSRD Follow Up 








4 


4 




3.3 Survey of SEA, LEA, and School 






y/ 




y 




4. Participation in the NEIA System 


y/ 


y/ 


y/ 


yf 


y 




5. Laboratory Network Program 


yf 


sj 


yf 


yf 


y 




5.1 Performance Indicators 


y/ 


y/ 


y/ 


yf 


y 




5.2 LNP Participation (cross-project) 




sj 


v/ 


yf 


y 




5.3 CLI Project 


y/ 


yf 


y/ 


y 






6. Assistance to OERI 


4 


4 


4 


4 


4 




6.1 Special Projects (to be determined) 














7. Specialty Area Development 


y/ 


y/ 


y/ 


y 


y 




7. 1 Special Studies (to be determined) 








4 


4 


III 


8. Collaboration 


y/ 


y/ 


y/ 








8.1 Regional Partners Meeting 


y/ 


4 


yf 


y 


y 




8.2 Survey of Partner 






y 


4 


y 
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LEVEL 


STUDY 


CONTRACT YEAR 
1 2 3 4 5 




9. Integrated Laboratory Evaluation 






9.1 Long-Term Impact 






9.2 Intensive Site Strategy Evaluation 






9.3 Scale-Up Strategy Evaluation 






Table notes: 

N New study, not included in original five-year plan 
p Postponed from previous contract year 
c Continued from previous contract year 
A As appropriate 



minutes and on interviews with the Laboratory Director, program staff, and field staff. The focus 
here is on describing and assessing the adequacy of the lines of communication among staff, the 
assignment of responsibility for service delivery, planning, reporting, quality control, and fiscal 
accountability. Recommendations for improving the management structure and operating 
procedures are developed based on input from staff. This study also examines the processes used 
to monitor regional educational needs and to adjust Laboratory services to be both responsive and 
timely. This study will be conducted annually. 

Board of Directors Survey (Study 1.1T This special study documents the manner by 
which the Board interacts and communicates with the laboratory staff. Recommendations for 
improving and clarifying the role of the McREL Board and their suggestions for strategic directions 
for the Laboratory are solicited in a survey of Board Members. These findings are integrated and 
summarized in conjunction with the overview of this area. 

Regional Needs Assessment (Study 1.2) . The purpose of this study of regional needs is 
to identify major educational issues of regional concern and to assess the level of support for public 
education. In Year 3 this study was based on telephone interviews with key influential educational 
leaders and households throughout the region, essentially replicating a survey conducted in 1 992/93. 
These interviews were conducted under a subcontract to the Gallup Organization, Inc. In addition, 
syntheses of national needs assessment findings will be undertaken on a periodic basis. The study 
will be replicated in Year 5 and findings used to help set the course for the next five years. 

Staff Perceptions of Organizational Functioning (Study 1.3) A survey of all McREL 
staff is conducted each year to assess their perceptions of the adequacy of McREL's organizational 
planning, external communication and services, internal decisions and communications, supervisory 
and peer relationships, and supporting functions. Staff are also asked to identify areas and strategies 
for improvement. The findings assist McREL's management council in mutually defining future 
directions for the Laboratory so that its work is consistent with and responsive to the identified needs 
of the our clients, avoids duplication of effort, and improves leveraging of OERI funds so that 
McREL can respond to a broader segment of the educational community across the region. 

Evaluation of Development and Applied Research (Study 2) 

This evaluation effort synthesizes the findings of the applied R&D activities, examines the 
utility of findings, and assesses the impact of McREL’s work on participants in intensive research 
sites. The evaluation addresses the four major R&D programs identified for this contract period: 
curriculum, instruction and assessment; human development, learning and motivation; organizational 
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curriculum, instruction and assessment; human development, learning and motivation; organizational 
learning and development; and systems integrations and systemic change. This study will be 
conducted annually. 

Capacity Building through Collaborative Research (Special Study 2. It Findings from 
the 1 998 Survey of Participants in Research Sites indicated that capacity building was the intended 
outcome most emphasized in McREL’s work with site participants. This study further defines what 
capacity building means in the intensive sites (i.e., what capacities did McREL researchers intend 
to address) and the impact the research work had on developing and improving those capacities. The 
special study assesses the extent to which McREL’s work in intensive sites has improved the 
individual, organizational and structural capacities of participants in their school districts (see 
Appendix for more information on the Task 2 special study). 

Evaluation of Field Services (Study 3) 

This overall evaluation of Task 3 integrates the assessments of various service interventions, state 
level activities, and educational topics outlined below. The evaluation is used to document the types 
of services provided, the clients reached, and area of focus for each service. As such, the 
information for this study comes from the activity reports (and associated database) previously 
described, records of McREL's resource center activities, data associated with electronic access to 
McREL via the Internet, and data collected directly from clients who participate in McREL's training 
and technical assistance activities. A generic client feedback form that can be tailored for individual 
service events is used to ensure that common data items are collected while meeting the individual 
project needs of the field services staff. 

Special studies are planned for each contract year in cooperation with the director of field 
services. These studies are implemented on a periodic basis using a combination of focus groups, 
individual interviews, and written questionnaires within the context of project-specific evaluations 
and, when appropriate, in cooperation with other RELs. 

Publications Evaluation Survey (Study 3.1) . Reader feedback was solicited through two 
written surveys distributed along with McREL's Changing Schools newsletter in Years 1 and 2. In 
Year 4 the evaluation focused on the perceived value of the newsletter, reactions to the format, and 
suggested topics for future issues in a series of focus groups. 

Assessment of Training Impacts (Study 3.2) . This special study is implemented as a series 
of independent studies with related themes. Follow-up mail and telephone surveys are conducted 
periodically with teachers and administrators who have worked with McREL on selected projects. 
For example, during this contract period, studies have focused on follow-up of the Achieving 
Excellence training provided to teachers and administrators in Nebraska and a follow-up study of 
participants who attended training on the NCES School Districts Database. Special studies planned 
for Year 5 include Special Study 3.2.7 which will focus on what has been learned about the quality 
of the Research into Practice Series facilitator/participant manuals from the user perspectives and 
the impact of the training on participants. Special study 3.2.8 will focus on the extent; quality, utility 
and impact of McREL’s contributions to support CSRD work in the region at both the state and local 
level. 



Survey of SEA, LEA, and School Administrators (Study 3.3) . This study proposed that 
telephone interviews be conducted with a representative sample of SEA, LEA, and school 
administrators during Years 3 & 5 of the contract period in order to assess their perceptions of 
McREL and the quality of its services and products, and to identify regional needs. McREL 
contracted with the Gallup Organization for this work because of its extensive experience and 
expertise in survey research and the organizationally sensitive nature of the data being collected. 
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Participation in NEIA System (Study 4) 



McREL collaborates with OERI, the other Regional Educational Laboratories, and the other 
participating technical assistance providers to conduct an evaluation of the process used to design 
the nationwide system as well as the design and effectiveness of the system itself. An independent 
evaluation plan, developed collaboratively, incorporates the set of benchmarks agreed upon against 
which progress and accomplishments will be measured. Study 4 examines McREL's facilitation of 
the system development and implementation in this region and will be conducted annually. 

Evaluation of Laboratory Network Program (Study 5) 

This overall evaluation task centers on an annual assessment of the major collaboration 
efforts undertaken during the contract period. Due to the relatively limited scope of this task, the 
associated analysis is primarily descriptive. Case study reports are developed for each collaborative 
effort undertaken. These summaries briefly describe the purpose of the major collaboration efforts 
and the extent to which each is successful in meeting its objectives and the goals set for the program. 

Development of Performance Indicators (Study 5.1) . Although not technically an 
evaluation project, development of the laboratory performance indicators has important implications 
for all levels of McREL's evaluation framework — from its evaluation infrastructure to cross- 
organizational work. The progress of this project and its relationship to indicators work in other 
related OERI work, such as the Eisenhower Math/Science Consortia, is closely monitored. McREL 
will continue to support the work of the REL evaluators group in the development, pilot testing, and 
use of performance indicators for the laboratory system on an annual basis. 

LNP Participation (Study 5.2) As LNP project plans continue to evolve under the direction 
of the lead laboratory in each specialty area, McREL’s participation also changes. McREL 
developed and implemented an internal cross-project evaluation to document the extent and nature 
of the Laboratory’ s participation and determine the perceived outcomes of these efforts on an annual 
basis beginning in Year 2. (Annual) 

CLI -- Project (Study 5.3) Each of the regional laboratories plays a lead role in the 
development and implementation of at least one LNP project in its specialty area. These studies 
share common themes of professional development, diversity, and national importance that are 
reflected in their individual evaluation plans. McREL’s study is a comparison of state approaches 
to standards-based educational reform. This study focuses on process evaluation, quality assurance, 
and outcomes of the applied R & D work. 

Assistance to OERI (Study 6) 



McREL conducts a variety of activities in response to requests from OERI. Evaluation of 
these activities is conducted annually as appropriate. 

Evaluation of Specialty Area Development (Study 71 

McREL’s specialty area under Task 7 of the OERI contract is curriculum, learning, and 
instruction. The role of the evaluation team with regard to Task 7 is seen as primarily supportive. 
The staff provides technical assistance to program staff, supports quality assurance activities, 
evaluates activities designed to further the Laboratory’s leadership role and examines the 
partnerships formed with other ED-funded institutions. Special event studies and/or special studies 
are planned as appropriate in cooperation with project directors. 
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Evaluation of Collaboration (Study 8) 



Collaboration is a theme that underlies much of McREL's planned OERI work. It is most 
clearly reflected in Tasks 2, 3, and 4 through the regional research team, regional field services team, 
and state facilitation groups which are comprised of state-level representatives from each of 
McREL's seven states. As such, it is important to examine the extent to which this collaboration is 
successful in contributing to the mission of the Laboratory. 

Regional Partners Meeting (Study 8.1) . (Annual). The Regional Partners Meeting 
represents the first major attempt to form new collaborative working groups in support of McREL's 
work. It is designed to bring together members of the state facilitation groups, regional research 
team, field services team, regional design team, SEA representatives, and various service provider 
groups in order to share information, establish roles and responsibilities, and agree on regional goals 
and objectives. 

Survey of Partners (Study 8.2) A survey of partners 1 , coordinated with data collection 
activities under Study 5, to assess the Laboratory's participation and collaboration in joint projects 
will be conducted in Years 3, 4 and 5. 

Integrated Laboratory Evaluation Study (Study 9) 

This assessment of Laboratory activities and their impact on the region's educational systems 
will be implemented as a synthesis of evaluation findings from relevant other studies in each of the 
three areas selected. Planning for the integrated study in FY 1999 led to the identification of three 
evaluation areas: Long Term Impact, Intensive Site Strategy and Scale Up Strategy. Findings will 
be incorporated into the Annual Evaluation Report at the end of the contract period. 



MANAGEMENT OF THE EVALUATION 

The OERI evaluation work is under the direction of Dr. Zoe Barley, who is supported by an 
evaluation team that is largely independent of the Laboratory's field service and applied research and 
development teams. An evaluation team member is assigned to each of the seven major contract 
tasks. The team member works with project/task directors and with other members of the evaluation 
team to implement and report on evaluation activities relevant to that task. In conjunction with 
McREL's Executive Director, general reporting and documentation requirements meet both 
evaluation and management information needs. Decisions regarding the selection of projects 
(service sites or R&D projects) for more in-depth evaluation are made jointly by the Executive 
Director, Dr. Barley, task leaders, evaluation coordinators, and the leaders of special projects. Due 
to the relatively distinct roles of service provider and evaluator established in its staffing plan, 
McREL does not anticipate the use of an external evaluation panel for most of the special studies 
planned. However, some circumstances do warrant the use of external evaluators. 

Use of External Evaluators 



McREL relies on external evaluators on an as-needed basis. At this time, several areas have 
been identified in which McREL believes the involvement of external evaluators is clearly necessary. 
In Years 3 and 4 McREL secured evaluation services to conduct a series of telephone surveys as part 



^ The survey instrument, developed by the cross-laboratory performance indicators task force for collecting feedback from ED- 
funded partners in joint ventures, is used with staff from those laboratories that are participating in the CLI-LNP project being led by McREL. 
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of the Laboratory's needs assessment and partnership building efforts (refer to special studies 1 .2 and 
8.2). In Year 5, external evaluators will be used to conduct studies 1.2 and 9.3. 

Reporting, Dissemination, and Utilization 

McREL maximizes the utilization of the evaluation findings by involving stakeholders in the 
collaborative evaluation approach, designing activities that are integral to ongoing task and project 
operations, and preparing and disseminating timely, constructive, and audience-specific information. 
The results and associated recommendations for all evaluation studies are reported to the appropriate 
task directors, involved staff, regional partner(s), and McREL's Executive Director. Evaluation 
Briefs are prepared for all special studies as well as Data Reports on participant feedback related to 
specific activities and a quarterly Activity Summaries for each Laboratory task and cost center. 
Quarterly progress reports are used to document evaluation activities and products across all 
Laboratory work. On an annual basis, a summary of McREL's evaluation efforts is reported in a 
separate Annual Evaluation Report that is distributed to OERI and presented to the governing board. 
This Annual Evaluation Report is designed to complement the Laboratory progress reports and 
emphasizes the findings and recommendations of the studies completed during the year. The scope 
and format of the report in years 4 and 5 is streamlined from that used in previous years to reduce 
duplication of effort, promote a more integrated approach to evaluation and program work, and 
enhance the timeliness of the report itself. 

As evaluation reports are available and key findings are documented, results are disseminated 
to the other regional laboratories and presented in periodic laboratory publications. The results of 
selected evaluations, particularly those related to successful demonstration projects which are 
suitable for replication in other locales, are published in professional journals, presented at selected 
educational conferences, and made available electronically through McREL's home page on the 
World Wide Web. 

Role of the Board of Directors 



The Board of Directors reviews evaluation results on at least an annual basis and makes 
recommendations regarding the need to modify or reorient Laboratory management, field services, 
or R&D practices and policies currently in place. The results of the needs assessment efforts 
conducted as Special Study 1.2 assist the Board in recommending new directions, identifying 
additional priority issues to be examined, or suggesting the reallocation of Laboratory resources 
among efforts already underway. 
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EVALUATION WORK PLAN 



The evaluation work consists of five essential activities: planning, improving the 
infrastructure, conducting overall task level studies, conducting special studies, and reporting. 
Specific deliverables, study schedules, staffing requirements, and budget breakdowns are described 
fully in annual work plans submitted to the OERI project director. 

• Planning 

Refine evaluation special studies for current FY 
Prepare detailed plans for special studies 
Prepare detailed plans for integrated studies 
Procure external contractors 

• Improve Evaluation Infrastructure 

Revise data collection forms and procedures 
Update supporting databases 
Enhance analysis capabilities 
Provide staff development 
Improve reporting formats 

• Conduct Task Level Evaluation 

1 . Laboratory Management 

2. Development and Applied Research 

3. Field Services 

4. Participation in the NEIA System 

5. Laboratory Network Program 

6. Assistance to OERI (project specific) 

7. Specialty Area Development 



Conduct Year 


5 Special Studies 


1.1 


Board Survey 


1.2 


Regional Needs Assessment 


1.3 


Staff Perceptions of Organization Functioning 


2.1 


Capacity Building through Collaborative Research 


3.2.7 


RIP Series Study 


3.2.8 


CSRD Follow Up 


5.1 


Performance Indicators 


5.2 


LNP Participation (cross-project) 


5.3 


CLI - Project 


6.1, 7.1 


(to be determined — as appropriate) 


8.1 


Regional Partners Meeting 


8.2 


Survey of Partners 


9.1 


Long Term Impact 


9.2 


Intensive Site Strategy Evaluation 


9.3 


Scale Up Strategy Evaluation 



• Reporting and Dissemination 

Prepare Annual Evaluation Report 

Prepare quarterly evaluation progress reports 

Prepare activity summaries for project directors 

Prepare internal data reports and project evaluation briefs 

Contribute to special project reports 

Participate in laboratory evaluator meetings 
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APPENDICES 



Special Study 1.1: 


Annual Survey of Board of Directors 


Special Study 1 .2: 


Regional Education Needs 


Special Study 1.3: 


Staff Perceptions of Organizational Functioning 


Special Study 2.1 : 


Capacity Building through Collaborative Research 


Special Study 3.2.7: 


RIP Series Quality and Impact Study 


Special Study 3.2.8: 


CSRD Follow Up 


Special Study 5.1 : 


REL Performance Indicators (FY99 and FYOO data) 


Special Study 5.3: 


CLI-LNP Project 


Special Study 9. 1 : 


McREL’s Long-Term Impact 


Special Study 9.2: 


Intensive Sites as a Strategy for Facilitating Reform 


Special Study 9.3: 


Scaling-Up/ An Assessment of Strategies 
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Evaluation Special Study Design 



Title: Special Study 1.1: Annual Survey of Board of Directors 

Task/Reference: Task 1.1 

Due Date: July 2000 

Reporting/Product: Not a deliverable. Results will be integrated into Task 1 report 

Focus of Study: 

The effectiveness of a Board of Directors is in part measured by their level of 
involvement in ongoing work. The McREL Board has been periodically surveyed or 
interviewed to determine their involvement in strategic decision-making for McREL. At the 
quarterly meeting of the Board in June of 2000, Directors will again be surveyed and/or 
interviewed. Their reflections on their roles as Board members and their perceived changes 
over the five years of this contract will provide a solid indicator of the Board’s engagement 
in strategic decision-making. The level of Board-staff interaction is a second indicator of 
Board involvement and providing direction to daily work. The Directors will also be asked 
to respond to their level of involvement in the ongoing work of McREL. 

Audiences: 

Board, Executive Director, Management Council, and Program and Project Directors. 

Evaluation Questions: 

1 . Do Board members feel they are sufficiently informed about McREL operations 
to participate in giving direction? 

2. Do Board members feel there is an enabling process to provide for input? 

3 . Do Board members perceive they make an important contribution to the direction 
McREL takes? 

4. Do staff and Board interact in a way that promotes effective use of both Board 
and staff expertise and regional connectedness? 

Methods: 

Board members will be asked to complete a survey during the June Board meeting. 

Connections to Other work: 

The results of the special study are incorporated in Study 1, the overview study of Task 1 of 
the REL contract. 
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Evaluation Special Study Design 



Title: Special Study 1 .2: Regional Education Needs 

Task/Reference: Task 1 

Due Date: April 2000 (postponed from 1999 to provide the appropriate time 

frame for a replication of an assessment done in 1 997). 

Reporting/Product: Deliverable 

Focus of Study: 

The purpose of this study is to determine from regional educators and state department 
personnel perceived needs for technical assistance, capacity building, training, networking, products, 
and tools. In addition, the study aims to determine from the same group their familiarity with 
McREL’s work. These findings will be used in shaping the next contract proposal to best meet the 
region’s needs. 

Audiences: 

OERI, regional colleagues, internal 

Evaluation Questions: 

1 . From the perspective of: teacher, principal, SDE staff, superintendent, or 
curriculum coordinator what capacities do you lack, what knowledge do you lack, 
what skills do you lack that would enable you to be more effective as an educator 
in the following areas? 



• Professional development 


• Comprehensive School Reform 


• Instructional strategies 


• T.iteracv 


• Standards-based education 


• Assessment 


• Parent/communitv involvement 


• Accountability 


• Forms of leadership 


• Technology 


_I Content knnwleHge/peHnpngy 





2. What is the familiarity of this group with McREL’s products and services? How 
do those who are familiar rate the quality? 

Methods: 

A contract has been given to The Gallup Organization who conducted the 1 997 study to 
replicate that study adding state department personnel. 

Connections to Other Work: 

Stand alone report. 
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Evaluation Special Study Design 



Title: Special Study 1.3: Staff Perceptions of Organizational Functioning 

Task/Reference: Task 1 

Due Date: August 2000 

Reporting/Product: Not a deliverable. Stand Alone Report 

Focus of Study: 

This study examines annually staff perceptions of their effectiveness, of the working 
environment at McREL, and of the resources and support they receive to accomplish their jobs are 
assessed. 

Audiences: 

McREL Supervisors, Board, and all staff 

Evaluation Questions: 

1. Do staff believe they are “developing and delivering high quality products and 
services”? 

2. Do staff perceive the McREL working environment as “positive and productive? 

3. Do staff have a comprehension and appreciation of organizational objectives and 
strategies? 

4. Are they focused collectively on effort and behaviors that accomplish 
organizational objectives? 

5. Have they identified and pursued individual or work group performance goals 
that support organizational objectives? 

6. Is there a clear “line of sight” between individual assignments and performance, 
and group performance and McREL success? 

Methods: 

A multi-part survey will be conducted during August. This survey is in part a continuation 
of surveys conducted in previous years permitting an analysis of trends in staff perceptions. An 
embedded research-based scale measuring the performance of the workplace is used as well as other 
items which combine a rubric with an open ended response format to obtain more qualitative data. 

Connections to Other Work: 

Stand alone report. 
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Evaluation Special Study Design 



Title: Special Study 2. 1 : Capacity Building through Collaborative Research 

Task/Reference: Task 2 

Due Date: September 2000 

Reporting/Product: Not a deliverable. Results will be presented as a stand-alone report 

Focus of Study: 

The special study for Applied Research and Development focuses on the impact of McREL’s 
work on long-term site participants and their school districts. The study will examine the extent to 
which McREL’s work has improved individual, organizational, and structural capacities. These 
include the Design Team members’ capacities to (a) design and conduct research for data-driven 
decision making, (b) take on new or improve extant research-related roles within the work 
environment, (c) collaborate intra- and inter-organizational ly, and (d) create and work within an 
organizational structure that promotes systemic reform. 

Audiences: 

Research team, site participants, Deputy Director 

Evaluation Questions: 

1. To what extent did Design Team members’ abilities to design and conduct 
research for data-driven decision making improve as a result of working with 
McREL? 

2. To what extent did Design Team members’ capacities to perform roles (e.g., 
teacher as researcher, learner, critical thinker, etc.) improve as a result of working 
with McREL? 

3 . Did McREL develop Design Team members’ capacities to collaborate with each 
other? 

4. Did McREL’s work impact Design Team members’ capacities to create and work 
within an organizational structure that promotes systemic reform? 

Methods: 

A literature review of capacity building in education will be conducted and will inform the 
development of a participant survey. The term capacity building also will be operationalized through 
discussions with researchers about their definitions of and intentions for capacity building in the 
intensive sites. The survey will assess the impact of McREL’s work on individual, organizational 
and structural capacities. On-site interviews will address the factors and conditions influencing these 
capacities and will explore other unintended impacts of working with McREL. Seventy-six Design 
Team members, including teachers, principals, counselors, district coordinators, and central 
administrators, representing five of McREL’s sites, will receive a survey which will be administered 
in early February 2000 and results compiled and analyzed to develop an interview protocol. On-site 
interviews at each site will be conducted March through May. Qualitative and quantitative data will 
be analyzed and reported in June 2000. 

Connections to Other work: 
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The results of this special study will be used in the Year 5 Lab report and the integrated 
study on intensive sites. 

Evaluation Special Study Design 



Title: Special Study 3.2.7: RIP Series Quality and Impact Study 

Task/Reference: Task 3 

Due Date: November 2000 

Reporting/Product: Not a deliverable. Stand alone report 

Focus of Study: 

McREL’s Field Service division is in the process of developing four topics supporting 
standards-based instruction as elements of the Research into Practice Series. Each topic includes 
a trainer’s manual and a participant manual, which will be available for dissemination throughout 
the region, once they have completed the pilot testing and quality review process. The Brief 
Overview of RIPs Field Test Model attached includes information about the development process. 
This study will report what has been learned about quality and utility of the training/manuals from 
trainer-user perspectives. It will also aggregate information from the expanded pilot testing about 
participant interests and accessibility of the necessary follow-up training. The field-test version to 
be developed after the expanded pilot process will be field tested - but not within this timeframe. 

Audiences: 

Stand alone report - internal use. 

Evaluation Questions: 

1 . What was learned from the initial sessions at McREL about needed changes in 
the materials/training? Were these findings enacted in the expanded pilot 
training/material? 

2. Was the expanded pilot process effective? Did the targeted group participate in 
each state? 

3 . What was learned from the expanded pilot about the training/materials in regards 
to the need for and access to additional training? 

4. flave the expanded pilot lessons been incorporated as the field test is launched? 

Methods: 

See attached Brief Overview 
Connections to Other work: 

This is a part of Task 3 evaluation. 
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Evaluation Special Study Design 

March 22, 2000 



Title: Special Study 3.2.8: CSRD Follow Up 

Task/Reference: Task (2), 3, & (5) 

Due Date: December 1999 

Reporting/Product: Not a deliverable. A stand-alone report will be generated. 

Focus of Study: 

McREL has received a second year of grant funding as an add-on to support CSRD work 
in the region and cross labs. These funds are folded into the current contract in some cases 
augmenting work in progress and in others identifying new work under Tasks 2, 3, and 5. No 
separate evaluation has been identified for the CSRD work. Because comprehensive school 
reform has been developed over long years in response to the need for improved student 
achievement and the inadequacies of less than comprehensive reform approaches, we believe it is 
a broad strategy in education that is here to stay. It will therefore be important for McREL to 
account for the quantity and quality of its contribution to CSR in the region. The focus of this 
special study is on the extent and impact of McREL’ s contributions through the work in Task 3 at 
the state and local level within the region. Task 2 research studies will generate their own reports 
that will be listed here as will Task 5 cross lab efforts. 

Audiences: 

McREL Management 

Evaluation Questions: 

1 . What services has McREL contributed to CSR at the state level?\ 

2. What services has McREL contributed to CSR at the LEA level? 

3. What has been the impact of these services? 

Methods: 

Document analysis of the field service notebooks will be conducted to identify services 
delivered to date. A format for recording services that augments the notebook material will be 
developed for use by McREL field service staff for future services. These two efforts will capture 
data on consultation or technical assistance other than training sessions, which are already recorded. 
For training sessions, the standard evaluation form will be enhanced to include information on prior 
LEA support from McREL and items assessing impact of prior and current support. A synthesis and 
analysis of the existing forms will permit an overview of client satisfaction with training. Syntheses 
from redeveloped forms will include impact analysis. SEA impact will be assessed through a series 
of structured interviews with the state coordinators for CSR. 

Connections to Other work: 

This work is embedded in the ongoing field service work but requires some modification to 
data collection to permit examining CSR separately; 
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Title: 



T ask/Reference: 
Due Date: 

Reporting Product: 



Evaluation Special Study Design 

February 2000 

Special Study 5.1 REL Performance Indicators 
(FY 1999 and FY 2000 data) 

Task 5.2 Performance Indicators 

Varies/January-December 2000 (dates tentative/TBD 

Continuation and expansion of McREL Performance Indicators Data 
for Cross-Laboratory Data Matrix (Deliverable #99-30 and #00-32); 
and Internal Feedback for McREL staff. 



Focus of Study: 

Under the Government Performance and Results Act of 1 993 and the scope of work for Task 
5.2 of the OERI contract, the RELs are required to submit cross-laboratory performance indicator 
data to OERI for inclusion in the Department of Education’s annual report to Congress. Indicators 
data are also used in an annual report generated by the RELs. The REL indicators address the 
following goal: 

To promote knowledge-based educational improvement to help all students meet high 
standards through development, applied research, dissemination, and technical 
assistance conducted with local, state, and intermediate agencies; 

and two specific objectives: 

a) field-based development and applied research and 

b) client services and product dissemination (see below). 

The ten RELs have formed a Cross-Laboratory Performance Indicators Group to guide and 
coordinate the collection, analysis, and reporting of indicators data. In 1 998, McREL staff used data 
from the laboratory’s activities database, internal data reports, and evaluation briefs as primary 
sources of information on products and services. McREL also adapted the indicator group’s generic 
surveys of partners and applied research sites by adding items to these instruments to provide 
additional evaluation feedback for internal use. In 1999, McREL continued these data-collection 
endeavors and began integrating the performance indicators into the laboratory’s ongoing, self- 
monitoring process. The performance indicators were revised by OERI in FY 1999, however, the 
Performance Indicators Group tentatively agreed to continue collecting data for all of the indicators 
for use in the REL annual report. In 2000, McREL will complete the data analysis and reporting for 
FY 1999 activities and refine the data collection as required for FY 2000 activities. As part of this 
effort, staff plan to strengthen the internal data infrastructure, collect additional follow-up data from 
clients on the quality, utility, and impact of laboratory services and products, and enhance the 
availability of formative evaluation feedback for McREL staff. 

Audiences: 

External 



McREL Indicators data submissions to Performance Indicators Group 
Performance Indicators Cross-laboratory Data Matrix to OERI, 
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REL Executive Directors, and Deputy Directors 

Cross-laboratory Indicators data submission to Communications Group 
for inclusion in Annual Report of the RELs 



Internal: 

McREL REL Task Leaders and Staff 

Evaluation Questions: (derived from the specific wording of the performance indicators) 

Objective 1. Field-Based Development and Applied Research : To what extent has McREL 
developed, adapted, and assessed comprehensive education reform strategies in schools, districts, 
and states? 

1.1 number of development sites (local or state sites engaged in collaborative 
development and demonstration of comprehensive reform-related efforts); 

1.2 quality of work at sites (percent of partners sampled rating work as contributing to 
comprehensive reform); 

1.3 moving reform to scale (number of new sites that apply what was learned at 
development sites); 

1.4 student impact (after 3 years at development sites, number or percent of sites showing 
increases in student achievement - - or interim/proxy measures TBD ) Required for FY 
2000; 

Objective 2. Client Services and Product Dissemination : T o what extent has McREL provided 
provided products and services and developed networks and partnerships in support of state and local 
reform? 



2.1 customer receipt of products and services (number of clients receiving 
products/services, web” hits”); 

2.2 quality of products and services (percent of clients sampled rating 
products/services as good/excellent); 

2.3 utility of products and services (percent of clients sampled reporting that 
products/services enhanced their knowledge/skills or professional practice, or were 
used for decision making); 

2.4 number of partnerships with practitioners, policymakers, and service 
providers; 

2.5 utility of partnerships (percent of partners/participants sampled reporting that 
partnerships addressed significant concerns and/or expanded their capacity for 
effective work); 
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Methods: 



McREL’s partners and clients/client organizations for FY 2000 REL contract work will be 
identified and record forms will be completed with the assistance of REL Task leaders and key 
program staff. Client impact data will be collected as part of McREL’s client satisfaction survey. 
An appropriate sampling frame will be determined. REL products and recipients of these products 
will be identified for FY 2000 with the assistance of McREL communications staff. The 
Laboratory’s activities database will also provide data regarding the delivery of services and 
products. 

Connections to Other work: 

As part of McREL’s ongoing self-monitoring process, this study entails the collection of data 
for McREL’s deliverable for Task 5.2 REL Performance Indicators. This activity is part of the work 
of the Cross-laboratory Performance Indicator Group. It also includes Surveys of Partners and 
Applied Research Sites as well as additional data that are collected as part of the evaluation for Task 
2, 3, 4, 5, and 7. It will also draw upon data collected for the integrated study on client satisfaction. 



Tentative Schedule for FY 1999/2000 Data * 





1999 


2000 


Collect data from 2000 McREL staff and clients 


12/99-01/00 


09/00-11/00 


Enter and analyze data (1/00-2/00) 


01/00-02/00 


11/00-12/00 


Submit cross-laboratory early indicators data to OERI 


01/00 


12/00 


Submit McREL data to REL Performance Indicator Group 
sub-group leaders 


01/00-02/00 


12/00 


Submit cross-laboratory data matrix to OERI 


03/00 


01/01 


Submit internal data reports to McREL task leaders 


06/00 


02/01 



*Note: Dates will be finalized by the Performance Indicator Group and OERI. 
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Evaluation Special Study Design 

February 2000 

Title: 

T ask/Reference: 

Due Date: 

Reporting/Product: 

Focus of Study: 

McREL is participating in the Laboratory Network Program (LNP), a national network 
linking the ten Regional Educational Laboratories (RELs) in a set of cross-laboratory projects. The 
LNP was established to strengthen the national laboratory system and to improve student 
achievement by translating education research into practice. The involvement of all of the 
laboratories is expected to provide the national perspective, to facilitate the completion of data 
collection requiring the cooperation of educational agencies nationwide, to support the development 
of products, and to promote the dissemination of findings to the regional laboratories and their 
constituents. 

The goals of this evaluation are to document and describe McREL's role, level of effort, and 
collaborative activities in the LNP projects; to assess the projects’ accomplishments to date; to 
identify factors that have affected project success; and to make recommendations for improving 
McREL's participation in the Program. The information will provide feedback that can be used to 
determine the value of McREL’s participation in the LNP and to help guide McREL’s future 
involvement and allocation of resources. 

Audiences: 

OERI, REL Program Leaders and Participating Staff, McREL Task Leaders and Staff 

Evaluation Questions: 

1. What is the nature/status of the LNP projects (e.g., design, activities, and 
intended outcomes)? 

2. What is the nature of collaboration among participating laboratories? 

3. What have been McREL’s commitments, levels of effort, and roles in the LNP? 

4. What has resulted from the LNP (e.g., products, dissemination, and 
understandings)? 

5. What are the major factors affecting the projects' success? 

6. How can McREL's participation in the LNP be improved? 

7. What lessons have been learned about participating in a collaborative cross- 
laboratory project? 

Methods: 

Information will be collected through a review of selected project documents and McREL’s 
quarterly reports; and brief surveys and/or interviews with the McREL staff members who are 
participants in the various projects. 

Connections to Other work: 



Special Study 5.2 Internal Study of McREL’s Participation in the LNP 
Task 5.1 
November 2000 
Evaluation Brief 
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This study will contribute to the overall Task 5 evaluation. 



Evaluation Special Study Design 

February 2000 

Title: Special Study 5.3 CLI-LNP Project 

Task/Reference: Task 5.1 

Due Date: Varies/Periodic Feedback and Data Reports 

Reporting/Product: Sections of Task 5 Study and Periodic Data Reports 

Focus of Study: 

McREL is participating in the Laboratory Network Program (LNP), a national network 
linking the ten Regional Educational Laboratories (RELs) in a set of cross-laboratory projects. The 
LNP was established to strengthen the national laboratory system and to improve student 
achievement by translating educational research into practice. The involvement of all of the 
laboratories is expected to provide the national perspective, to facilitate the completion of data 
collection requiring the cooperation of education agencies nationwide, to support the development 
of products, and to promote the dissemination of findings to the regional laboratories and their 
constituents. As part of the LNP, McREL is leading a collaborative research and development (R 
& D) project in its specialty area of Curriculum, Learning, and Instruction (CLI). This national 
descriptive study focuses on the impacts of state curriculum-based reform policy in districts and 
schools in terms of student learning and educational practice. The research study is designed in 
phases, with data collection progressing from the state level to the district and school levels. 

The project is expected to result in a series of products including reports on each level of the 
study and guidelines for reform implementation. Products may also include electronic reports, other 
tools, and presentations of research findings at selected conferences. The target audiences include 
educators and policy makers who have a direct interest in the success of school reform, state 
legislatures, state education personnel, district and school administrators, classroom teachers, 
professional associations, and REL staff who provide services to support educational reform within 
their regions. The research findings are expected to promote an understanding of the complexities 
and challenges that confront educational agencies engaged in systemic reform. Laboratory staff are 
expected to benefit from a broader perspective on curriculum-based reform as a context in which to 
develop technical assistance to support regional reform efforts. 

The purposes of the evaluation are to a) provide formative evaluation feedback about 
collaboration that can be used to improve the project as it develops; b) establish/promote quality 
assurance procedures to support the development of high quality products; c) determine if the 
project is successful in achieving intended outcomes and impacts; and d) identify lessons learned 
about conducting a collaborative R & D project. 

Audiences: 

OERI, REL Program Leaders and Participating Staff, McREL Task Leaders and Staff 
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Evaluation Questions: 

1 . What is the nature of the project (e.g., design, processes, activities, intended 
outcomes)? 

2. What is the nature of collaboration among participating laboratories (e.g., 
roles, obstacles encountered and how these have been addressed, lessons 
learned)? 

3. What has resulted from the R&D activities (e.g., products, dissemination, 
understandings, and utility)? 

4. What lessons have been learned about conducting a collaborative R&D 
project? 



Methods: 

The evaluation will be conducted in phases corresponding to the research study design and 
will include the use of document review, photographs, participant evaluation feedback (e.g., process 
evaluation 

worksheets, structured group activities, surveys of project staff in relation to outcomes), quality 
assurance guidelines, product dissemination records, and feedback from target audiences on selected 
CLI products. 

Connections to Other work: 

This study will contribute to the Task 5 evaluation and REL indicators work. 
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Evaluation Integrated Study Design 

February 8, 2000 



Title: Integrated Study 9.1 : McREL’s Long-Term Impact 

Task/Reference: All Tasks - 

Due Date: October 2000 

Reporting/Product: Not a deliverable. Stand Alone Report 

Focus of Study: 

This study seeks to answer the question — what has been the impact of McREL’ s work during 
the current contract? 

Audiences: 

McREL Management, Board, and staff 

Evaluation Questions: 

1 . What have been the long-term effects of McREL’s work in this contract period? 

2. What capacities have been developed among clients? 

3. What use of products has occurred and how have they influenced client 
work?What new knowledge/leamings do clients report from trainings? 

4. What research/policy results have been disseminated to contribute to the wider 
body of educational knowledge? 

Methods: 

This integrated study compiles information from all available sources to assess the long-term 
impact of McREL’s work during the current contract. Analysis and compilation of data and results 
from other studies is a key method. Some new data collection may be required to fill gaps in the data 
set. 

Connections to Other work: 

Draws upon all other available relevant work. 
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Evaluation Integrated Study Design 

February 8, 2000 

Title: Integrated Study 9.2: Intensive Sites as a Strategy for Facilitating Reform 

Task/Reference: Tasks 2 & 3 

Due Date: October 2000 

Reporting/Product: Not a deliverable. Stand Alone Report 

Focus of Study: 

In order to facilitate systemic reform in McREL’s region, researchers and field service staff 
have established long-term collaborative partnerships with LEAs as intensive sites. In the research 
sites, the joint work centered on collaborative action research involving a Design Team at each site. 
At the field service sites the nature of work varied with the context and needs of the LEA. This 
study seeks to answer the question - what has been the impact of McREL’s work during the current 
contract? 

Audiences: 

McREL Management, Board, and staff 

Evaluation Questions: 

1 . What have been the long-term effects of McREL’ s work in this contract period? 

2. What capacities have been developed among clients? 

3. What use of products has occurred and how have they influenced client work? 

4. What new knowledge/leamings do clients report from trainings? 

5. What research/policy results have been disseminated to contribute to the wider 
body of educational knowledge? 

Methods: 

This integrated study compiles information from all available sources to assess the long-term 
impact of McREL’s work during the current contract. Analysis and compilation of data and results 
from other studies is a key method. New data collection is required for the field service sites - 
research sites are being visited under Task 2. Each long-term site will receive a two-day visit from 
a team of two evaluators. Interviews will be conducted with administrators, teachers will be 
surveyed, documents reviewed, and observations made as indicated. 

Connections to Other work: 

Draws upon all other available relevant work. 
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Evaluation Integrated Study Design 



February 8, 2000 



Tas k/Reference : 



Title: 



Integrated Study 9.3: Scaling Up: An Assessment of Strategies 
All Tasks - 



Due Date: 



October 2000 



Reporting/Product: Not a deliverable. Stand Alone Report 

Focus of Study: 

This study reviews a variety of scale up strategies McREL has used during the present 
contract including several that emerged more recently. Older strategies are judged for effectiveness, 
more recent ones are assessed for potential as a continuing strategy. 

Audiences: 

McREL Management, Board, and staff 

Evaluation Questions: 

1. What has been the nature of McREL’ s scale-up strategies during the current 
contract? 

2. Has McREL been responsive to changing patterns in educational practice in 
shaping scale-up strategies? 

3. Do the scale-up strategies in use reflect both a concern for breadth and depth? 

4. Has the strategy of working with intensive sites proved effective? 

5. Has the strategy of working with SEAs proved effective? 

6. Has the strategy of serving clients through a website proved effective? 

7. Has the strategy of doing round tables in states proved effective? 

8. Does the more recent strategy of the Research into Practice Series hold promise 
as a strategy? 

9. Does the more recent strategy of the consortium of districts hold promise as a 
strategy? 

10. How effective overall has McREL’ s scale-up strategy been? 



This integrated study combines work on one special study, 2.1, with new work on the 
intensive field service sites conducted by McREL’s evaluation unit. In addition subcontracts will 
be let to provide analysis of the Research into Practice Series, the Standards Consortium, the SEA 
work, and the use of round tables as scale up strategies. An integrated report addressing the entire 
set of scale-up strategies will be produced. 

Connections to Other Work: 

Draws upon all other available relevant work. 



Methods: 
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